Function-space inference with sparse Implicit Processes

S. Rodríguez-Santana, B. Zaldívar, D. Hernández-Lobato

39th International Conference on Machine Learning - ICML 2022, Baltimore (United States of America). 17-23 July 2022

Summary:

Implicit Processes (IPs) represent a flexible framework that can be used to describe a wide variety of models, from Bayesian neural networks, neural samplers and data generators to many others. IPs also allow for approximate inference in functionspace. This change of formulation solves intrinsic degenerate problems of parameter-space approximate inference concerning the high number of parameters and their strong dependencies in large models. For this, previous works in the literature have attempted to employ IPs both to set up the prior and to approximate the resulting posterior. However, this has proven to be a challenging task. Existing methods that can tune the prior IP result in a Gaussian predictive distribution, which fails to capture important data patterns. By contrast, methods producing flexible predictive distributions by using another IP to approximate the posterior process cannot tune the prior IP to the observed data. We propose here the first method that can accomplish both goals. For this, we rely on an inducing-point representation of the prior IP, as often done in the context of sparse Gaussian processes. The result is a scalable method for
approximate inference with IPs that can tune the prior IP parameters to the data, and that provides accurate non-Gaussian predictive distributions.

Published in Proceedings of Machine Learning Research, vol: 162, pp: 18723-18740

Publication date: 2022-07-23.

Citation:
S. Rodríguez-Santana, B. Zaldívar, D. Hernández-Lobato, Function-space inference with sparse Implicit Processes, 39th International Conference on Machine Learning - ICML 2022, Baltimore (United States of America). 17-23 July 2022. In: Proceedings of Machine Learning Research, vol. 162, e-ISSN: 2640-3498